Image processing apparatus, image processing method, and program

ABSTRACT

An image processing apparatus includes: an image input section through which a two-dimensional image signal is input; an image converting section which receives the image signal output from the image input section and generates and outputs a left eye image and a right eye image for realizing stereopsis; and an image output section which outputs the left eye image and the right eye image output from the image converting section. The image converting section extracts a spatial characteristic amount of the input image signal, generates at least one of the left eye image and the right eye image through an image conversion process in which the characteristic amount is applied, and performs the image conversion process of a different type according to a comparison result of distance information of a preset region unit in the input image signal and a predetermined threshold.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method and a program, and more particularly, to an imageprocessing apparatus, an image processing method and a program whichperforms image conversion for a two-dimensional image to generate abinocular disparity image corresponding to stereopsis.

2. Description of the Related Art

In the related art, a variety of proposals has been made for apparatusesand methods which convert a two-dimensional image into a binoculardisparity image corresponding to stereopsis. The binocular disparityimage generated on the basis of the two-dimensional image includes apair of a left eye image, which is observed by the left eye, and a righteye image, which is observed by the right eye. As the binoculardisparity image including the pair of the left eye image and the righteye image is displayed through a display apparatus which can separatethe binocular disparity image into the left eye image and the right eyeimage to respectively provide the separated images to the left eye andthe right eye of an observer, the observer can perceive the images as athree-dimensional image.

Such an image generation or display process is disclosed in thefollowing related art techniques.

For example, Japanese Unexamined Patent Application Publication No.9-107562 discloses an image processing configuration for a dynamic imagemoving horizontally. Specifically, according to this configuration, anoriginal image is output as one of a left eye image and a right eyeimage, and an image delayed in a field unit is output as the otherthereof. Through such an image output control, an object moving in thehorizontal direction is perceived as being in front of the background.

Further, Japanese Unexamined Patent Application Publication No. 8-30806discloses an apparatus in which a left eye image and a right eye imageare shifted from each other by a preset amount in the horizontaldirection for a still image or an image having a small movement and thusthe image is perceived as floated.

Further, Japanese Unexamined Patent Application Publication No. 10-51812discloses a technique in which an image is divided into a plurality ofdisparity calculation regions, a pseudo-depth is calculated from acharacteristic amount of the image in each region, and a left eye imageand a right eye image are horizontally shifted in the oppositedirections on the basis of the depth.

Further, Japanese Unexamined Patent Application Publication No.2000-209614 discloses a technique similar to Japanese Unexamined PatentApplication Publication No. 10-51812, which while changing, on the basisof a delay amount calculated from a characteristic amount of an image,the horizontal delay amount of a left eye image and a right eye image,limits the horizontal delay amount to prevent a retinal image differencefrom occurring any more than is necessary, thereby preventing a degreeof fatigue of the eyes.

Further, Japanese Unexamined Patent Application Publication No.2005-151534 discloses a technique in which characteristic amounts ofupper and lower portions of an image are calculated, and the synthesisratio of a plurality of scene structures indicating preset depthinformation is adjusted, to thereby express the image as a combinationhaving a simplified structure.

However, the related art techniques as described above have thefollowing problems.

In the image conversion apparatus as disclosed in Japanese UnexaminedPatent Application Publication No. 9-107562, preferable stereopsis canbe provided for only an object moving at a constant speed in thehorizontal direction. In an image including a plurality of movingobjects or complex movements, the binocular disparity is not correctlyset, and thus the objects are arranged in unnatural positions or theretinal image difference becomes excessively large. Thus, stereopsis maynot be formed.

Further, the image conversion apparatus as disclosed in JapaneseUnexamined Patent Application Publication No. 8-30806 provides only theshift of the entire screen for the still image or the image having asmall movement, but has great difficulty in expressing theanteroposterior relation of the objects in the image.

In the image conversion apparatuses as disclosed in Japanese UnexaminedPatent Application Publication Nos. 10-51812 and 2000-209614, thepseudo-depth is estimated from the characteristic amounts of the image,but since the estimation is based on such an assumption that thesharpness, luminance and saturation of the object in the front of thescreen are high, the estimation is not necessarily performed in acorrect manner. Thus, an incorrect retinal image difference is given tothe object having the incorrect depth estimation, thereby resulting inthe arrangement in an incorrect position.

The image conversion apparatus as disclosed in Japanese UnexaminedPatent Application Publication No. 2005-151534 is configured to allowthe structure of the image to be adapted for a simple and limitativestructure and suppresses generation of unnatural depth. However, all therelated art techniques as described above have the following commonproblem. That is, a relatively large retinal image difference isgenerated in the generated binocular disparity image, and the binoculardisparity image is three-dimensionally displayed using athree-dimensional display apparatus. In general, such athree-dimensional display apparatus is used that the image is observedusing special glasses for stereopsis, such as a passive eyeglass-type inwhich an image is divided through a polarization filter, or a colorfilter to be observed by the left eye and the right eye or an activeeyeglass type in which an image is temporally divided from side to sidethrough a liquid crystal shutter.

When viewing the binocular disparity image having the large retinalimage difference, an observer can perceive the stereoscopic effectdepending on the retinal image difference with such glasses forstereopsis. However, when viewing the screen without the glasses, theobserver comes to view double images in which the left eye image and theright eye image overlap with each other by a large amount and does notview the images as a normal two-dimensional image. That is, the imageconverted through these related art image conversion apparatuses can beviewed only with the glasses.

Further, the large retinal image difference may affect fatigue of theobserver. For example, Japanese Unexamined Patent ApplicationPublication No. 6-194602 discloses that in a case where the left eyeimage and the right eye image are each shifted by a large amount, adiscrepancy occurs between control of the angle of convergence andadjustment of the lens in a view direction in the real world, so thatthe discrepancy leads to fatigue in stereopsis using the binoculardisparity.

Further, in the image conversion apparatuses as disclosed in JapaneseUnexamined Patent Application Publication Nos. 10-51812, 2000-209614 and2005-151534, the pseudo-depth is estimated from the image, but it isdifficult to detect a detailed depth from a single image. For example,in the case of minute structures such as branches of trees, electriccables or hairs, the depth estimation is not easy. Thus, it is difficultto obtain the stereoscopic effect for minute objects.

SUMMARY OF THE INVENTION

Accordingly, it is desirable to provide an image processing apparatus,an image processing method and a program which can suppress generationof an incorrect stereoscopic effect due to an incorrect depth estimationto restore an original image or an image close to the original imagewhen a left eye image and a right eye image are synthesized. That is,the present invention provides an image processing apparatus, an imageprocessing method and a program in which an observer can view a normaltwo-dimensional image without glasses for stereopsis and which cangenerate and provide a binocular disparity image, reducing fatigue forthe observer.

According to an embodiment of the invention, there is provided an imageprocessing apparatus including: an image input section through which atwo-dimensional image signal is input; an image converting section whichreceives the image signal output from the image input section andgenerates and outputs a left eye image and a right eye image forrealizing stereopsis; and an image output section which outputs the lefteye image and the right eye image output from the image convertingsection, wherein the image converting section is configured to extract aspatial characteristic amount of the input image signal and to generateat least one of the left eye image and the right eye image through animage conversion process in which the characteristic amount is applied,and performs the image conversion process of a different type accordingto the comparison result of distance information of a preset region unitin the input image signal and a predetermined threshold.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section is configured to extract aluminance differential signal of the input image signal, to set theluminance differential signal as the characteristic amount, to generateone of a converted signal obtained by adding the characteristic amountto the input image signal or a converted signal obtained by subtractingthe characteristic amount from the input image signal as one of the lefteye image and the right eye image, and to output a non-converted signalfrom the input image signal without being processed as the other one ofthe left eye image and the right eye image.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section is configured to extract aluminance differential signal of the input image signal, to set theluminance differential signal as the characteristic amount, to generatea signal obtained by adding the characteristic amount to the input imagesignal and a signal obtained by subtracting the characteristic amountfrom the input image signal, and to generate a set of the two signals asa set of the left eye image and the right eye image.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section is configured to extract aluminance differential signal of the input image signal, to set a signalgenerated by non-linearly converting the luminance differential signalas the characteristic amount, to generate a signal obtained by addingthe characteristic amount to the input image signal or a signal obtainedby subtracting the characteristic amount from the input image signal,and to generate one of the generated signals as the left eye image orthe right eye image.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section is configured to switch theprocess of generating the left eye image or the right eye image byadding the characteristic amount to the input image signal and theprocess of generating the left eye image or the right eye image bysubtracting the characteristic amount from the input image signal,according to the comparison result of the distance information of thepreset region unit in the input image signal and the predeterminedthreshold.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section is configured to generatethe left eye image as a signal obtained by subtracting thecharacteristic amount from the input image signal and generate the righteye image as a signal obtained by adding the characteristic amount tothe input image signal in a case where the relation between the distanceof the preset region unit in the input image signal and thepredetermined threshold satisfies “distance≦threshold”, and to generatethe left eye image as the signal obtained by adding the characteristicamount to the input image signal and generate the right eye image as thesignal obtained by subtracting the characteristic amount from the inputimage signal in a case where the relation satisfies“distance>threshold”.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section switches an addition processand a subtraction process between the input image signal and thecharacteristic amount according to the distance information of thepreset region unit and generates the left eye image and the right eyeimage of which the perception range is enlarged to anteroposteriorregions of a display section.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section is configured to generatethe left eye image and the right eye image for each frame forming amoving image.

Further, in the image processing apparatus according to an embodiment ofthe invention, the apparatus further includes the image output sectionwhich outputs the left eye image and the right eye image generated bythe image converting section, wherein the image output section isconfigured to alternately output the left eye image and the right eyeimage generated by the image converting section at a speed two timesfaster than an input image frame rate.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section is configured to alternatelygenerate only either one of the left eye image or the right eye imagefor each frame forming a moving image.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section is configured to generatethe left eye image and the right eye image for each frame forming amoving image and to generate a binocular disparity image alternatelyincluding line data forming the generated left eye image and right eyeimage.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image converting section is configured to generatethe left eye image and the right eye image so that an addition signal ofthe generated left eye image and right eye image is set to be equivalentor approximately equivalent to the input signal.

Further, in the image processing apparatus according to an embodiment ofthe invention, the apparatus further includes an image display sectionwhich displays the images generated by the image converting section.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image display section is configured to perform athree-dimensional display process using a time-division type in whichthe left eye image and the right eye image are alternately output.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image display section is configured to performdisplay switching so that an output switching timing of the left eyeimage and the right eye image is synchronized with the shutter switchingof left and right glasses sections of glasses which an image observerwears, when performing the three-dimensional display process using thetime-division type in which the left eye image and the right eye imageare alternately output.

Further, in the image processing apparatus according to an embodiment ofthe invention, the image display section has a configuration in which apolarizing filter which is set so that a polarization direction becomesdifferent for every horizontal line is installed on a front surface ofthe display section, and is configured to display a binocular disparityimage alternately including line data forming the left eye image and theright eye image generated by the image converting section.

Further, according to an embodiment of the invention, there is providedan image processing apparatus including: an image input section throughwhich a two-dimensional image signal is input; an image convertingsection which receives the image signal output from the image inputsection and generates and outputs a left eye image or a right eye imagefor realizing stereopsis; and an image output section which outputs theleft eye image and the right eye image output from the image convertingsection, wherein the image converting section is configured to generatethe left eye image and the right eye image so that an addition signal ofthe generated left eye image and right eye image is set to be equivalentor approximately equivalent to the input signal, and performs an imageconversion process of a different type according to the comparisonresult of distance information of a preset region unit in the inputimage signal and a predetermined threshold.

Further, according to an embodiment of the invention, there is providedan image processing method in an image processing apparatus, includingthe steps of: inputting a two-dimensional image signal, through an imageinput section; receiving the image signal output from the image inputsection and generating and outputting a left eye image and a right eyeimage for realizing binocular stereopsis, by an image convertingsection; and outputting the left eye image and the right eye imageoutput from the image converting section, through an image outputsection, wherein the step performed by the image converting sectionincludes: extracting a spatial characteristic amount of the input imagesignal; generating at least one of the left eye image and the right eyeimage through an image conversion process for performing an emphasisprocess in which the characteristic amount is applied to the input imagesignal; and performing the image conversion process of a different typeaccording to the comparison result of distance information of a presetregion unit in the input image signal and a predetermined threshold.

Further, according to an embodiment of the invention, there is provideda program which allows image processing to be executed in an imageprocessing apparatus, including the steps of: enabling an image inputsection to receive an input of a two-dimensional image signal; enablingan image converting section to receive the image signal output from theimage input section and generating and outputting a left eye image and aright eye image for realizing binocular stereopsis; and enabling animage output section to output the left eye image and the right eyeimage output from the image converting section, wherein the step ofenabling the image converting section includes: extracting a spatialcharacteristic amount of the input image signal; generating at least oneof the left eye image and the right eye image through an imageconversion process for performing an emphasis process in which thecharacteristic amount is applied to the input image signal; andperforming the image conversion process of a different type according tothe comparison result of distance information of a preset region unit inthe input image signal and a predetermined threshold.

Here, the program according to the embodiment includes a program whichcan be provided through a storage medium or a communication medium whichis provided in a computer-readable format to a general-use system whichcan execute a variety of program codes, for example. By providing such aprogram in a computer-readable format, a process corresponding to theprogram is realized on the computer system.

Other objects, features and advantages of the invention will be apparentthrough the following detailed description based on the embodiments ofthe invention and the accompanying drawings. In this specification, theterm “system” refers to a configuration in which a plurality of devicesor apparatuses is logically assembled, and is not limited to aconfiguration in which all the component devices or apparatuses aredisposed within the same casing.

According to the configuration in the embodiments of the invention, thetwo-dimensional image signal is input to generate the left eye image andthe right eye image for realizing the binocular stereopsis. The imageconverting section extracts the spatial characteristic amount of theinput image signal, and generates the left eye image and the right eyeimage by performing the different emphasis process in which thecharacteristic amount is applied to the input image signal.Specifically, the luminance differential signal of the input imagesignal or the signal generated by non-linearly converting the luminancedifferential signal is used as the characteristic amount. The set of thesignal generated by adding the characteristic amount to the input imagesignal and the signal generated by subtracting the characteristic amountfrom the input image signal is generated as the set of the left eyeimage and the right eye image. Further, the addition process and thesubtraction process is switched according to the distance information ofthe pixel unit or the preset region unit so as to enlarge the perceptionrange to the anteroposterior regions of the display section. With thisconfiguration, it is possible to generate images capable of stereopsiswith a simple signal processing. In addition, since the addition signalof the left eye image and the right eye image is equivalent to the inputimage signal, it is possible to observe the images as a normaltwo-dimensional image when observing the images without glasses forstereopsis.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration example of an imageprocessing apparatus according to an embodiment of the presentinvention.

FIG. 2 is a diagram illustrating a configuration example of an imageinput section of an image processing apparatus according to anembodiment of the present invention.

FIG. 3 is a flowchart illustrating a process sequence in a case where aninput image is a still image, which is a process example of an imageinput section of an image processing apparatus according to anembodiment of the present invention.

FIG. 4 is a flowchart illustrating a process sequence in a case where aninput image is a moving image, which is a process example of an imageinput section of an image processing apparatus according to anembodiment of the present invention.

FIG. 5 is a diagram illustrating a configuration example of an imageconverting section of an image processing apparatus according to anembodiment of the present invention.

FIG. 6 is a diagram illustrating an example of a non-linear conversionprocess for an image signal performed by an image converting section ofan image processing apparatus according to an embodiment of the presentinvention.

FIG. 7 is a diagram illustrating a configuration example of an imagesynthesizing section in an image converting section of an imageprocessing apparatus according to an embodiment of the presentinvention.

FIG. 8 is a diagram illustrating an example of an image signalgenerating process for the right eye and the left eye from an inputimage performed by an image converting section of an image processingapparatus according to an embodiment of the present invention.

FIG. 9 is a diagram illustrating an example of an image signalgenerating process for the right eye and the left eye from an inputimage performed by an image converting section of an image processingapparatus according to an embodiment of the present invention.

FIG. 10 is a diagram illustrating an example of an image signalgenerating process for the right eye and the left eye from an inputimage performed by an image converting section of an image processingapparatus according to an embodiment of the present invention.

FIG. 11 is a diagram illustrating an example of an image signalgenerating process for the right eye and the left eye from an inputimage performed by an image converting section of an image processingapparatus according to an embodiment of the present invention.

FIG. 12 is a diagram illustrating an example of an image signalgenerating process for the right eye and the left eye from an inputimage performed by an image converting section of an image processingapparatus according to an embodiment of the present invention.

FIG. 13 is a diagram illustrating a retinal image difference between aright eye image and a left eye image generated by an image processingapparatus according to an embodiment of the present invention.

FIG. 14 is a diagram illustrating a retinal image difference between aright eye image and a left eye image generated by an image processingapparatus according to an embodiment of the present invention.

FIG. 15 is a diagram illustrating a retinal image difference between aright eye image and a left eye image generated by an image processingapparatus according to an embodiment of the present invention.

FIG. 16 is a diagram illustrating a retinal image difference between aright eye image and a left eye image generated by an image processingapparatus according to an embodiment of the present invention.

FIGS. 17A and 17B are diagrams illustrating a disparity setting and aperceived image between a right eye image and a left eye image.

FIG. 18 is a diagram illustrating a disparity setting and a perceivedimage between a right eye image and a left eye image.

FIG. 19 is a diagram illustrating an example of an image signalgenerating process for the right eye and the left eye from an inputimage performed by an image converting section of an image processingapparatus according to an embodiment of the present invention.

FIGS. 20A, 20B and 20C are diagrams illustrating a disparity setting anda perceived image between a right eye image and a left eye image.

FIG. 21 is a flowchart illustrating a process sequence performed by animage converting section of an image processing apparatus according toan embodiment of the present invention.

FIG. 22 is a flowchart illustrating a generating process sequence of aleft eye image performed by an image converting section of an imageprocessing apparatus according to an embodiment of the presentinvention.

FIG. 23 is a flowchart illustrating a generating process sequence of aright eye image performed by an image converting section of an imageprocessing apparatus according to an embodiment of the presentinvention.

FIG. 24 is a diagram illustrating a configuration example of an imageprocessing apparatus according to an embodiment of the presentinvention.

FIG. 25 is a diagram illustrating an example of a non-linear conversionprocess for an image signal performed in an image converting section ofan image processing apparatus according to an embodiment of the presentinvention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, an image processing apparatus, an image processing methodand a program according to the present invention will be described withreference to the accompanying drawings.

FIG. 1 is a diagram illustrating an image processing apparatus accordingto an embodiment of the present invention. The image processingapparatus 100 receives a still image file output from a digital stillcamera or the like, or moving image data output from a camcorder or thelike through an image input section 110, and converts the received fileor data into an internal data format. Here, the internal data formatrefers to baseband moving image data, which is video data of threeprimary colors of red (R), green (G) and blue (B) or video data ofluminance (Y) and color difference (Cb, Cr). The internal data formatmay have any color space as long as color space identifying signals areoverlapped and a color space converting section 2 in a subsequent stagecorresponds thereto.

The video data output from the image input section 110 is input to thecolor space converting section 120, and is converted into a luminancesignal and a color difference signal. At this time, in a case where theinput video data is based on Y, Cb and Cr color spaces, the color spaceconverting section 120 outputs the video data without conversion intothe color spaces. In a case where the input video data is based on theR, G and B color spaces, or other color spaces, the color spaceconverting section 120 converts the video data into the luminance (Y)and color difference (Cb, Cr) signals for output.

In this respect, the color spaces of the video data output from thecolor space converting section 120 are not limited to the Y, Cb and Crspaces, but may be any spaces as long as the color spaces are colorspaces in which a luminance component and a color component areseparated from each other.

The video data output from the color space converting section 120 isinput to an image converting section 130. The image converting section130 generates a binocular disparity image for a left eye and a right eyethrough a process to be described later, and synthesizes and outputs theimages according to the format of a three-dimensional display apparatus.That is, the image converting section 130 extracts a spatialcharacteristic amount of an input image signal, and generates a left eyeimage and a right eye image by performing a different emphasis processfor the extracted characteristic amount.

The video data output from the image converting section 130 is input toa color space reverse converting section 140, and is converted intocolor spaces based on the output image format from the Y, Cb and Crcolor spaces. At this time, in a case where the output image format isbased on the Y, Cb and Cr color spaces, the color space reverseconverting section 140 outputs the video data without conversion intothe color spaces. The image processing apparatus in FIG. 1 has theconfiguration of the color space converting section 120 and the colorspace reverse converting section 140, but this configuration is notnecessary, and thus may be omitted.

The video data output from the color space reverse converting section140 is input to the image output section 150. The image output section150 converts and outputs the color spaces into video data which isreceivable in the externally connected three-dimensional displayapparatus which can display the binocular disparity image converted inthe image converting section 130 so as to realize stereopsis.

In this embodiment, in a case where the still image is input, the methodof converting the still image into the video data in the image inputsection 110 is disclosed, but the present invention is not limitedthereto. For example, one still image may be converted into two imagesfor left eye image and right eye image, and two still images may beoutput to a memory card or the like as a file, for example.

FIG. 2 is a block diagram illustrating a configuration of the imageinput section 110 according to an embodiment. The image input section110 is provided with a memory card interface 111 for input of a stillimage file or the like; a USB interface 112 for direct connection with avideo device; a video interface 113 for input of a video signal; a framememory 114; a decoder 115; and a video output section 116.

As an example of a process in the image input section 110, a processsequence in a case where a still image is input will be described withreference to a flowchart shown in FIG. 3.

In step S101, the image input section 110 starts receiving an input ofthe still image.

In step S102, the image input section 110 confirms whether a memory cardis inserted in the memory card interface 111, and determines whetherimage data is input from the memory card. In a case where the memorycard is inserted, the procedure goes to step S104, and in a case wherethe memory card is not inserted, the procedure goes to step S103.

In step S103, the image input section 110 confirms whether an externaldevice which can input a still image is connected to the USB interface112, and determines whether image data is input through the USBinterface 112. In a case where the USB device is connected, theprocedure goes to step S105, and in a case where the USB device is notconnected, the image input process is terminated.

Here, in order to determine which medium receives the input of movingimage data, a method of giving an instruction to an input device using amanipulation section (not shown) may be used.

In step S104, the image input section 110 reads the image data from thestill image file recorded in the memory card. At this time, the stillimage file in the memory card may be selected using the manipulationsection (not shown), or may be automatically selected according to theorder determined according to a specific standard.

In step S105, the image input section 110 reads the still image datafrom the external device connected to the USB interface. At this time,the still image file in the external device may be selected using themanipulation section (not shown), or may be automatically selectedaccording to the order determined according to a specific standard.

In step S106, the image input section 110 stores the still image dataread in step S104 or step S105 in a frame memory 114.

In step S107, the image input section 110 reads the still image datafrom the frame memory 114 by a controller (not shown). At this time, theread address represents a head of the image data stored in step S106.

In step S108, the image input section 110 decodes the still image. Sincethe still image data is generally image-compressed according to a formatregulated according to the JPEG (Joint Photographic Experts Group) orthe like, the decoder 115 performs an image decompression processaccording to the image format, and restores baseband image data.

In step S109, the image input section 110 outputs the decoded stillimage data as one frame of the video data. Here, the format of the videodata is based on an output format in the image output section 150. Thatis, in a case where video data having a HD (High Definition) resolutionand a frame rate of 60 frames per second is output from the image outputsection 150, the controller (not shown) generates a videosynchronization signal having a HD resolution a frame rate of 60 framesper second, and pastes the still image in an effective region of thesignal for output.

In step S110, it is determined whether the image output process in theimage output section 150 is terminated. In a case where the image outputprocess is terminated, the image input process is terminated. In a casewhere the image output process is not terminated, the procedure goes tostep S111.

In step S111, the image input section 110 initializes a reading addressof the frame memory 114 and displays the head of the still image datastored in step S106. If the address initiation in step S111 isterminated, the procedure goes to step S107, and then the processes ofstep S107 to step S111 are repeated.

In this way, in the case where the still image is input, the image inputsection 110 converts the still image into video data in which the sameimages are continuous.

Next, as an example of a process in the image input section 110, aprocess sequence in a case where the moving image is input will bedescribed with reference to a flowchart shown in FIG. 4.

In step S201, the image input section 110 starts receiving an input ofthe moving image.

In step S202, the image input section 110 confirms whether a videosignal is input through the video interface 113, and determines whethermoving image data is input through the video interface 113. In a casewhere the video signal is input, the procedure goes to step S205, and ina case where the video signal is not input, the procedure goes to stepS203.

In step S203, the image input section 110 confirms whether an externaldevice from which a moving image can be input is connected to the USBinterface 112, and determines whether the moving data is input throughthe USB interface 112. In a case where the USB device is connected, theprocedure goes to step S206, and in a case where the USB device is notconnected, the procedure goes to step S204.

In step S204, the image input section 110 confirms whether the memorycard is inserted in the memory card interface 111, and determineswhether moving image data is input from the memory card. In a case wherethe memory card is inserted, the procedure goes to step S207, and in acase where the memory card is not inserted, the image input process isterminated.

Here, in order to determine whether the moving image data is input froma specific data medium, a method of giving an instruction to the inputapparatus using the manipulation section (not shown) may be used.

In step S205, the image input section 110 reads the video data from thevideo interface 113. The video interface 113 receives an input of avideo signal transmitted in a digital video transmission method such asDVI (Digital Video Interface), HDMI (High-Definition MultimediaInterface), HDSDI (High-Definition Serial Digital Interface) or thelike, or a video signal transmitted in an analog video transmissionmethod such as the NTSC (National Television Standards Committee)method, a component method or the like. In a case where an analog videosignal is input, the video interface 113 converts the input analog videosignal to a baseband signal by a demodulation process, and then convertsthe baseband signal into a digital signal by an A/D converter (notshown). On the other hand, in a case where a digital video signal isinput, the digital video signal is converted into a baseband signal bythe demodulation process.

In step S206, the image input section 110 reads the moving image datafrom the external device connected to the USB interface 112. At thistime, the moving image file in the external device may be selected usingthe manipulation section (not shown), or may be automatically selectedaccording to the order determined according to a specific standard.

In step S207, the image input section 110 reads the image data from themoving image file recorded in the memory card. At this time, the movingimage file in the memory card may be selected using the manipulationsection (not shown), or may be automatically selected according to theorder determined according to a specific standard.

In this respect, the moving image data input through the USB interface112 or the moving image data recorded in the memory card is streamingdata compressed with a moving image compression method regulatedaccording to the MPEG (Moving Picture Experts Group or the like). Sincesuch a compression method should perform a decoding process using theframe memory, the stream data is stored in the frame memory 114 in stepS208.

In step S209, the image input section 110 reads the moving image datafrom the frame memory 114 by the controller (not shown).

In step S210, the image input section 110 decodes the moving image. Asdescribed above, since the moving image data stored in the frame memory114 is the stream data compressed through the MPEG or the like, thedecoder 115 performs the image decompression process based on the imageformat and restores the baseband video data.

In step S211, the video output section 116 outputs any one of the videodata output from the video interface 113 and the video data output fromthe decoder 115 in the internal data format.

FIG. 5 is a block diagram illustrating a configuration of the imageconverting section 130 according to an embodiment. The image convertingsection 130 extracts the spatial characteristic amount of the inputimage signal, and generates the left eye image and the right eye imageby performing the different emphasis process for the extractedcharacteristic amount. The image converting section 130 includes adifferentiator 131, a non-linear converting section 132 and an imagesynthesizing section 133. The differentiator 131 extracts a luminancesignal from the video data input to the image converting section 130,and then generates a differential signal for the luminance signal.Specifically, for example, the luminance signal of the image is input inthe horizontal direction, and the differential signal is generated byperforming a first-order differentiation for the input luminance signal.In the first-order differentiation process a linear first-orderdifferential 3-tap transversal filter is used, for example.

The non-linear converting section 132 non-linearly converts thedifferential signal output from the differentiator 131, and outputs theconverted signal as a disparity emphasis signal.

FIG. 6 is an example of a non-linear conversion process performed in thenon-linear converting section 132. The horizontal axis represents aninput signal from the differentiator 131, which is a luminancedifferential signal. The vertical axis represents an output after thenon-linear conversion process in the non-linear converting section 132.The non-linear converting section 132 converts an input differentialsignal (In) according to a predetermined function f(x) to output adisparity emphasis signal (Out). That is, Out=f(In). Here, the functionf(x) can be set in a variety of manners. As an example of the functionf(x), for example, an exponential function as shown in the followingformula can be used,

f(x)=x ^(β)

where β is a preset coefficient, which can be set to a variety ofvalues.

Further, the conversion function in the non-linear converting section132 is not limited to the exponential function, and may use a linearconversion.

FIG. 25 illustrates another example of the non-linear conversion processperformed in the non-linear converting section 132. In a similar way toFIG. 6, the horizontal axis represents an input signal from thedifferentiator 131, which is a luminance differential signal. Thevertical axis represents an output after the non-linear conversionprocess in the non-linear converting section 132. The non-linearconverting section 132 converts an input differential signal (In)according to a predetermined function f(x) to output a disparityemphasis signal (Out). That is, Out=f(In). The example shown in FIG. 25is an example in a case where a function f(x) shown in the followingformula 10 is used as the function f(x). Here, gain, γA, γB and Da arepredetermined coefficients, which can be set as a variety of values.

$\begin{matrix}{{{if}\mspace{14mu} \left( {{in} \leqq {Da}} \right)\text{:}}{{out} = \left( {{in} \times {gain}} \right)^{\gamma \; A}}{{if}\mspace{14mu} \left( {{in} > {Da}} \right)\text{:}}{{out} = {\left\{ {\left( {1.0 - {in}} \right) \times {gain}} \right)^{\gamma \; B} \times \frac{\left( {{Da} \times {gain}} \right)^{\lambda \; A}}{\left\{ {\left( {1 - {Da}} \right) \times {gain}} \right\}^{\gamma \; B}}}}} & \left( {{formula}\mspace{14mu} 10} \right)\end{matrix}$

Such a non-linear conversion process is performed to generate a moredesirable three-dimensional image. For example, in the non-linearconversion process shown in FIG. 25, if the input differential signal(In) exceeds an arbitrary threshold Da, the disparity emphasis signal(Out) tends to be restricted. With such an effect, it is possible torestrict an unnecessary high frequency emphasis of the right eye imagesignal and left eye image signal which are finally generated, and alsoto emphasize a depth feeling of the right eye image signal and left eyeimage signal which are generated.

The image synthesizing section 133 receives the disparity emphasissignal output from the non-linear converting section 132 and the videodata input to the image converting section 130, and synthesizes eachframe image and the disparity emphasis signal for forming the video datato generate a left eye image and a right eye image.

As indicated by a dashed line in FIG. 5, the image converting section130 may have a configuration in which the conversion process of thenon-linear converting section 132 is omitted and the differential signalgenerated by the differentiator 131 is directly input to the imagesynthesizing section 133, and thus, the image synthesizing section 133generates the left eye image and the right eye image using thedifferential signal.

In this way, the image synthesizing section 133 generates the left eyeimage and the right eye image using the differential signal generated bythe differentiator 131 or the disparity emphasis signal output from thenon-linear converting section 132, as the spatial characteristic amountof the input image signal.

The image synthesizing section 133 generates the left eye image and theright eye image using each frame image for forming the video data, andthe spatial characteristic amount generated from the frame image, thatis, the differential signal of the luminance signal or the disparityemphasis signal generated by nonlinearly converting the differentialsignal.

FIG. 7 is a block diagram illustrating a configuration of the imagesynthesizing section 133 according to an embodiment. The imagesynthesizing section 133 includes an adder 134, a subtractor 135, adistance information (distance image) generating section 136 and aselector 137.

The distance information (distance image) generating section 136calculates or obtains the distance from the photographing position(camera position) in pixel units for forming an image which is a processtarget, or in preset region units of a block or the like, including apredetermined plurality of pixels, and outputs the result to theselector 137. Specifically, for example, the distance information(distance image) generating section 136 generates a distance image inwhich the distance is indicated as a luminance level and outputs thedistance image to the selector 137. Existing techniques set out in therelated art can be used as a method of obtaining or generating thedistance of the image forming pixel unit or the preset region unit. Forexample, distance information measured by a distance measurement devicesuch as a laser-range scanner at the time of image generation is addedto the image as image attribute information, to thereby obtain thedistance. Alternatively, the distance of the pixel or the preset regionunit is calculated using existing techniques for calculating thedistance through image analysis.

The distance information obtained or generated by the distanceinformation (distance image) generating section 136 is output to theselector 137. The selector 137 compares a preset distance threshold withthe distance of a pixel (distance from a camera) in a current process,which is output from the distance image generating section 136.According to the comparison result, it is determined whether a left eyeimage (L image) and a right eye image (R image) is to be output througheither the adder 134 or the subtractor 135, for output.

The adder 134 or the subtractor 135 adds or subtracts the disparityemphasis signal obtained by nonlinearly converting the differentialsignal to or from the input signal, for output. That is, the adder 134generates and outputs a signal of “input signal+disparity emphasissignal”, and the subtractor 135 generates and outputs a signal of “inputsignal−disparity emphasis signal”.

The selector 137 compares the preset distance threshold with thedistance of the pixel (distance from the camera) in the current processwhich is output from the distance image generating section 136, andoutputs the output of the adder 134 and the output of the subtractor 135as the left eye image (L image) or the right eye image (R image),according to the following table 1.

TABLE 1 distance from photographing position output image outputdistance ≦ threshold left eye image input signal − disparity (L image)emphasis signal (output of subtractor) right eye image input signal +disparity (R image) emphasis signal (output of adder) Distance >threshold left eye image input signal + disparity (L image) emphasissignal (output of adder) right eye image input signal − disparity (Rimage) emphasis signal (output of subtractor)

As shown in the Table 1, the image synthesizing section 133 sets andoutputs the left eye image (L image) and the right eye image (R image)as follows, according to the distance information from the distanceimage generating section 136.

In the case of “distance from the photographing position≦threshold”, theleft eye image (L image) is “input signal−disparity emphasis signal”(output of the subtractor), and the right eye image (R image) is “inputsignal+disparity emphasis signal” (output of the adder), which areoutput as the left eye image (L image) and the right eye image (Rimage), respectively.

On the other hand, in the case of “distance from the photographingposition>threshold”, the left eye image (L image) is “inputsignal+disparity emphasis signal” (output of the adder), and the righteye image (R image) is “input signal−disparity emphasis signal” (outputof the subtractor), which are output as the left eye image (L image) andthe right eye image (R image), respectively.

An example of the image synthesis process performed by the imagesynthesizing section 133 will be described with reference to FIG. 8. Theprocess shown in FIG. 8 is a process corresponding to a pixel region ina case where the distance from the photographing position in the abovetable 1 satisfies “distance>threshold”. In this case, the imagesynthesizing section 133 generates the left eye image (L image) which is“input signal+disparity emphasis signal” (output of the adder) and theright eye image (R image) which is “input signal−disparity emphasissignal” (output of the subtractor), and outputs the left eye image (Limage) and the right eye image (R image), respectively.

FIG. 8 illustrates an input signal (a), a differential signal (b), aright eye image signal (c) and a left eye image signal (d),respectively, in the order named from the top.

Here, the input signal (a) represents change in the luminance of asingle arbitrary horizontal line in an arbitrary frame of the videodata. There is exemplified one line in which a high luminance regionhaving a high level of luminance exists in the center area thereof. In aregion A ranging from a line position (x1) to a line position (x2), theluminance is changed to gradually become high. Between the line position(x2) and a line position (x3), the high luminance region in which thehigh level of luminance is maintained exists. Then, in a region Branging from the line position (x3) to a line position (x4), theluminance is changed to gradually decrease.

The differential signal (b) is a result obtained by differentiating theinput signal (a). The differential signal is a signal generated in thedifferentiator 131 of the image converting section 130 shown in FIG. 5.

As shown in the figure, the differential signal generated in thedifferentiator 131 has a positive value in the region A in which theluminance change of the input signal (a) becomes positive, and has anegative value in the region B in which the luminance change of theinput signal (a) becomes negative.

The right eye image signal (c) and the left eye image signal (d) aresignals generated in the image synthesizing section 133 of the imageconverting section 130 shown in FIG. 5. The image synthesizing section133 synthesizes the input signal (a) with the disparity emphasis signalwhich is the result (output of the non-linear converting section 132)obtained by non-linearly converting the differential signal (b) in thenon-linear converting section 132, to generate the right eye imagesignal (c) and the left eye image signal (d).

It is assumed that the luminance level of the video data correspondingto the input signal (a) in FIG. 8 is (S), and the signal level of thedisparity emphasis signal obtained by non-linearly converting thedifferential signal (b) in FIG. 8 is (E).

The image synthesizing section 133 receives the video data (S)corresponding to the input signal (a) and the disparity emphasis signal(E) obtained by non-linearly converting the differential signal (b), andgenerates the right eye image signal (Right) and the left eye imagesignal (Left) according to the following formula 1, for example.

Right=S−E

Left=S+E  (formula 1)

Here, the image synthesizing section 133 may perform conversion for anyone signal, without conversion of both of the left eye image signal(Left) and the right eye image signal (Right) as shown in the formula 1.

That is, the following signal combination may be available.

Right=S−E

Left=S

Alternatively, the following signal combination may be available.

Right=S

Left=S+E

Through such a process, images in which a retinal image difference isgenerated and depth is perceived, can be obtained as the right eye imagesignal (Right) and the left image signal (Left). Further, the relationbetween the retinal image difference and the depth perception will bedescribed with reference to FIG. 13.

As described above, the image converting section 130 may have aconfiguration in which the conversion process of the non-linearconverting section 132 is omitted and the differential signal generatedby the differentiator 131 is directly input (see the dashed line in FIG.5) to the image synthesizing section 133, and then, the imagesynthesizing section 133 generates the left eye image and the right eyeimage using the differential signal. In this case, the disparityemphasis signal (E) is replaced with the differential signal.

In this way, the image synthesizing section 133 generates the left eyeimage and the right eye image by extracting the spatial characteristicamount of the input image signal and performing a different emphasisprocess in which the characteristic amount is applied to the input imagesignal. The characteristic amount is the luminance differential signalof the input image signal or the disparity emphasis signal generated bythe non-linear conversion process for the luminance differential signal.

The right eye image signal (Right) (c) in FIG. 8 is a signal generatedby subtracting the disparity emphasis signal (E) obtained bynon-linearly converting the differential signal (b) from the inputsignal (a).

The right eye image signal (Right) (c) is generated as a signal havingthe following signal characteristics (c1) to (c3) as shown in the righteye image signal (c) in FIG. 8.

(Signal Characteristics)

(c1) In at least a part of the region A in which the luminance change ofthe input signal (a) is a positive value and the differential signal (b)is a positive value, a signal region having luminance lower than that ofthe input signal (a) occurs.

(c2) In at least a part of the region B in which the luminance change ofthe input signal (a) is a negative value and the differential signal (b)is a negative value, a signal region having luminance higher than thatof the input signal (a) occurs.

(c3) In a region where the differential signal (b) is a value “zero”,the luminance change for the input signal (a) does not occur.

Further, the left eye image signal (Left) (d) in FIG. 8 is a signalgenerated by adding the disparity emphasis signal (E) obtained bynon-linearly converting the differential signal (b) to the input signal(a).

The left eye image signal (Left) (d) is generated as a signal having thefollowing signal characteristics (d1) to (d3) as shown in the left eyeimage signal (d) in FIG. 8.

(Signal Characteristics)

(d1) In at least a part of the region A in which the luminance change ofthe input signal (a) is a positive value and the differential signal (b)is a positive value, a signal region having luminance higher than thatof the input signal (a) occurs.

(d2) In at least a part of the region B in which the luminance change ofthe input signal (a) is a negative value and the differential signal (b)is a negative value, a signal region having luminance lower than that ofthe input signal (a) occurs.

(d3) In a region where the differential signal (b) is a value “zero”,the luminance change for the input signal (a) does not occur.

As described above, the image synthesizing section 133 synthesizes theinput signal (a) with the disparity emphasis signal which is the result(output of the non-linear converting section 132) obtained bynon-linearly converting the differential signal (b) in the non-linearconverting section 132, to generate the right eye image signal (c) andthe left eye image signal (d).

The image synthesizing section 133 generates, for example, if the inputsignal which is a conversion target corresponds to a still image, theright eye image signal (c) and the left eye image signal (d) through thesignal synthesis process based on the formula 1, for one frame image forforming the still image.

If the input signal which is the conversion target corresponds to amoving image, the image synthesizing section 133 generates the right eyeimage signal (c) and the left eye image signal (d) through the signalsynthesis process based on the formula 1, for each frame for forming themoving image. Here, in the case of the moving image, it is possible tochange the method of generating the right eye image signal and the lefteye image signal, according to the image output section 150 (see FIG. 1)for finally performing the image display or the control method of thedisplay apparatus. Hereinafter, in a case where the input signal whichis the conversion target is the moving image (video data), a pluralityof exemplary processes performed by the image synthesizing section 133will be described with reference to FIG. 9.

Firstly, in the case where the input signal which is the conversiontarget is the moving image (video data), an exemplary basic processperformed by the image synthesizing section 133 will be described withreference to FIG. 9. In the process example shown in FIG. 9, the imagesynthesizing section 133 generates and outputs both images of the lefteye image (Left) and the right eye image (Right), for all frames (framen, n+1, n+2, n+3 . . . ) of the input video data.

The image synthesizing section 133 synthesizes a luminance signal of aninput image frame (a) with a disparity emphasis signal which is theresult obtained by non-linearly converting a differential image signal(b), for all frames of the input image (a) shown in FIG. 9, to generatea right eye image signal (c) and a left eye image signal (d) in FIG. 9.In this case, the image synthesizing section 133 outputs two types ofvideo signals.

The synthesis process is performed according to the above formula 1, forexample. That is, when it is assumed that the luminance level of thevideo data corresponding to the input signal (a) in FIG. 8 is (S) andthe signal level of the disparity emphasis signal obtained bynon-linearly converting the differential signal (b) shown in FIG. 8 is(E), the right eye image signal (Right) and the left eye image signal(Left) are generated according to the following formula.

Right eye image signal: Right=S−E

Left eye image signal: Left=S+E

In the basis process example shown in FIG. 9, the image synthesizingsection 133 outputs two types of video signals of the right eye imageand the left eye image corresponding to all the frames. The image outputsection 150 (see FIG. 1) receives the two types of signals and outputsdata thereon to a display apparatus for realizing stereopsis. Thedisplay apparatus performs the output control according to a variety ofdisplay methods for realizing stereopsis. As the display methods of thedisplay apparatus, for example, there is an image output methodcorresponding to a passive eyeglass type in which an image is dividedthrough a polarization filter or a color filter to be observed by theright eye and the left eye, or an image output method corresponding toan active eyeglass type in which an image is alternately and temporallydivided by alternately opening and closing a liquid crystal shutter fromside to side to be observed by the left eye and the right eye. Thedisplay apparatus displays the images based on any display method asdescribed above using the two types of video signals generated by theimage synthesizing section 133.

In a case where the image display method is determined in advance, theimage synthesizing section 133 can be set to generate and output anoutput image signal according to each image output method. Hereinafter,process examples of the image synthesizing section 133 according tothree different display methods will be described with reference toFIGS. 10 to 12.

There are three types of display methods of the display apparatus forfinally performing the image display, as follows.

(1) Method of alternately outputting the left eye image and the righteye image in a time division manner (FIG. 10)

This is an image output method corresponding to the active eyeglass typein which the image is divided by alternately opening and closing aliquid crystal shutter from side to side in a temporally alternatingmanner, for example, to be observed by the right eye and the left eye.

(2) Method of alternately outputting the left eye image and the righteye image in a time division manner, in which an output frame rate ishigh-speed (FIG. 11).

This method uses the same time division as in FIG. 10, in which theoutput frame rate is high-speed.

(3) Method of spatially dividing the left eye image and the right eyeimage for a simultaneous output (FIG. 12).

This is an image output method corresponding to the passive eyeglasstype in which the image is divided through a polarization filter or acolor filter, for example, to be viewed by the right eye and the lefteye, respectively. For example, in the three-dimensional displayapparatus of the space division type, in a case where a polarizationfilter, which is set so that the polarizing direction becomes differentfor every horizontal line, is installed on the front display surface anda user views the images using glasses of the polarizing filter type, theimages are separated for every horizontal line and are observed by theleft eye and the right eye.

A process example of the image synthesizing section 133 in a case wherethe display apparatus employs the display method of alternatelyoutputting the left eye image and the right eye image in a time divisionmanner to finally perform the image display, will be described withreference to FIG. 10.

In the case of the image display method, the image synthesizing section133 alternately generates the left eye image (Left) and the right eyeimage (Right) for each frame (frame n, n+1, n+2, n+3, and so on) of theinput video data, for output.

An odd frame and an even frame of the input video data are set andoutput as the left eye image and the right eye image (or right eye imageand left eye image), respectively. In the output image, the left eyeimage and the right eye image in the image display apparatus arealternately output in a time division manner through the image outputsection 150. The output timing of each image is controlled to besynchronized with the opening and closing of shutters of liquid crystalshutter-type glasses, which the user wears to observe the images, forexample. That is, the output timing is controlled so that the left eyeimage and the right eye image are alternately and temporally observed bythe left eye and the right eye, respectively.

Since the output is performed through a three-dimensional displayapparatus of such a time division type, the image synthesizing section133 performs the image synthesis process for each frame (frame n, n+1,n+2, n+3, and so on) of the input video data by switching the left eyeimage and the right eye image in a frame unit. That is, as shown in (c)and (d) in FIG. 10, the synthesis of the left eye images (Left) and thesynthesis of the right eye images (Right) are alternately performed inthe frame unit, for output.

In the examples shown in FIG. 10, the right eye image is generated inthe frame n according to the above formula 1. That is, when theluminance level of the video data in the frame n of the input signal (a)in FIG. 10 is (S) and the signal level of the disparity emphasis signalobtained by non-linearly converting the differential signal (b) of theframe n in FIG. 10 is (E), the right eye image signal (Right) isgenerated according to the following formula.

Right eye image signal: Right=S−E

In the next frame n+1, the left eye image is generated according to theformula 1. That is, when the luminance level of the video data in theframe n+1 of the input signal (a) in FIG. 10 is (S) and the signal levelof the disparity emphasis signal obtained by non-linearly converting thedifferential signal (b) of the frame n+1 in FIG. 10 is (E), the left eyeimage signal (Left) is generated according to the following formula.

Left eye image signal: Left=S+E

Here, the right eye image is generated in the frame n+2, and the lefteye image is generated in the frame n+3. Thereafter, similarly, theright eye image and the left eye image are alternately generated andoutput for every frame through the image synthesis process according tothe formula 1. In this method, the image synthesizing section 133generates and outputs one of the right eye image and the left eye imagein correspondence with each frame. That is, one type of video data isoutput.

The output images are alternately output as the left eye image and theright eye image in the image display apparatus in a time divisionmanner, through the image output section 150. The output timing of eachimage is controlled to be synchronized with the opening and closing ofthe shutters of the liquid crystal shutter-type glasses which the userwears to observe the images. That is, the output timing is controlled sothat the left eye image and the right eye image are alternately andtemporally observed by the left eye and the right eye, respectively.

FIG. 11 is a process example of the image synthesizing section 133 in acase where the display apparatus employs the display method ofalternately outputting the left eye image and the right eye image in atime division manner to finally perform the image display, in a similarway to the process in FIG. 10. Here, in this process, both of the lefteye image (Left) and the right eye image (Right) are synthesized throughthe synthesis process according to the formula 1 for each frame of theinput video data for output, unlike the process shown in FIG. 10.

In the display apparatus for performing the image output, the left eyeimage and the right eye image are alternately output in a time divisionmanner at a frame rate which is two times higher than the input videodata.

In this process, the image synthesizing section 133 generates a righteye image (c) and a left eye image (d) from a disparity emphasis signalwhich is generated from one frame, for example, a frame n of an inputimage (a) and a differential image (b), using the formula 1, as shown inFIG. 11. Then, the image synthesizing section 133 generates the righteye image (c) and the left eye image (d) from a disparity emphasissignal which is generated from the next frame, that is, a frame n+1 ofthe input image (a) and the differential image (b), using the formula 1.

In this way, the left eye image and the right eye image are generatedfrom one frame. Two images generated from one frame, that is, the righteye image and the left eye image are alternately output as the left eyeimage and the right eye image in the image display apparatus in a timedivision manner, through the image output section 150.

The image output section 150 outputs the images to be displayed at aframe rate which is two times higher than the frame rate of the inputimage (a) shown in FIG. 11 in the display apparatus. The opening andclosing of the shutters of the liquid crystal shutter-type glasses, forexample, which the user wears to observe the images is controlled to besynchronized with the display timing of each image. That is, it iscontrolled so that the left eye image and the right eye image arealternately and temporally observed by the left eye and the right eye.In this method, the image synthesizing section 133 outputs the videodata at the frame rate which is two times higher than the one type inputvideo data.

FIG. 12 is a process example of the image synthesizing section 133 in acase where the output is performed to the three-dimensional displayapparatus of the space division type. In the three-dimensional displayapparatus of the space division type, in a case where a polarizingfilter which is set so that the polarizing direction becomes differentfor every horizontal line is installed on a front display surface and auser views images using glasses of the polarizing filter type, theimages are separated for every horizontal line and provided to the lefteye and the right eye. That is, polarization filters of the right andleft sides of the glasses are also set to have different polarizationdirections. Thus, only a right eye image (c) shown in FIG. 12 isobserved by the right eye, and only a left eye image (d) shown in FIG.12 is observed by the left eye.

In this process, as shown in FIG. 12, the image synthesizing section 133generates the right eye image (c) and the left eye image (d) from adisparity emphasis signal generated from one signal, for example, aframe n of the input signal (a) and a differential signal (b), using theformula 1.

Further, the image synthesizing section 133 generates a binoculardisparity image (e) shown in FIG. 12 from the right eye image (c) andthe left eye image (d). That is, each image of the right eye image (c)and the left eye image (d) is reduced to ½ in the vertical direction byshifting the phase thereof by one line. The image synthesizing section133 alternately synthesizes the left eye image and the right eye imageobtained in this manner in the horizontal line unit to generate andoutput a single binocular disparity image (d).

The binocular disparity image (d) shown in FIG. 12 is an image generatedby connecting effective regions (image display sections other than blacklines) of the right eye image (c) and the left eye image (d). That is,the binocular disparity image (d) alternately includes each line data ofthe right eye image (c) and the left eye image (d). The imagesynthesizing section 133 generates and outputs the binocular disparityimage (d) in this way. In this method, the image synthesizing section133 outputs one type video data at the same frame rate as in the inputimage.

The image output section 150 outputs the binocular disparity image (d)shown in FIG. 12 to the three-dimensional display apparatus of the spacedivision type to be displayed. As described above, in thethree-dimensional display apparatus of the space division type, thepolarizing filter which is set so that the polarizing direction becomesdifferent for every horizontal line is installed on the front displaysurface thereof. A user observes the images using the glasses of thepolarizing filter type. The polarizing filters on the right and leftsides of the glasses are also set to have different polarizationdirections. Thus, only right eye image (c) shown in FIG. 12 is observedby the right eye, and only left eye image (d) shown in FIG. 12 isobserved by the left eye.

The right eye image signal (Right) and the left eye image signal (Left)described with reference to FIGS. 9 to 12 are generated according to theabove formula 1. That is, the right eye image signal (Right) and theleft eye image signal (Left) are generated according to the followingformula.

Right=S−E

Left=S+E

Here, S represents an input signal, and E represents a disparityemphasis signal obtained by non-linearly converting a differentialsignal D of the input signal S. Further, as described before, thedisparity emphasis signal E may be a signal obtained using a linearconversion, in place of the non-linear conversion of the differentialsignal D of the input signal S.

The right eye image signal (Right) and the left eye image signal (Left)generated in this manner are observed by the right eye and the left eyeof an observer, so that the observer can sense depth. This is aphenomenon based on a retinal image difference between the right eyeimage and the left eye image. The retinal image difference between theright eye image and the left eye image generated in the image processingapparatus 100 according to the present invention will be described withreference to FIGS. 13 to 16. Hereinafter, in FIGS. 13 to 15, it isassumed that the non-linear conversion process for the differentialsignal D is omitted and the right eye image signal (Right) and the lefteye image signal (Left) are generated using the input signal S and thedifferential signal D of the input signal S according to the followingformula.

Right=S−D

Left=S+D

FIG. 13 is a diagram illustrating a retinal image difference generatedby addition or subtraction of the differential signal. For simplicity ofthe description, FIG. 13 shows a state where a left eye signal and aright eye signal is generated in a case where a one-dimensional sinewave signal is input as the input signal. The horizontal axis in thefigure represents a pixel position in the image in the horizontaldirection and the vertical axis represents a luminance level of thepixel.

The input signal S is expressed as the following formula (formula 2).

S=sin ωx  (formula 2)

Here, the differential signal D is expressed as the following formula(formula 3).

D=cos ωx  (formula 3)

At this time, the left eye signal L and the right eye signal R areexpressed as the following formulas (formula 4 and formula 5).

$\begin{matrix}{L = {{S + D} = {{{\sin \; \omega \; x} + {\cos \; \omega \; x}} = {\sqrt{2}{\sin \left( {{\omega \; x} + \frac{\pi}{4}} \right)}}}}} & \left( {{formula}\mspace{14mu} 4} \right) \\{R = {{S - D} = {{{\sin \; \omega \; x} - {\cos \; \omega \; x}} = {\sqrt{2}{\sin \left( {{\omega \; x} - \frac{\pi}{4}} \right)}}}}} & \left( {{formula}\mspace{14mu} 5} \right)\end{matrix}$

From the formula 4 and formula 5, the left eye signal L has a phasewhich advances from the input signal S by n/4, and the right eye signalR has a phase which delays from the input signal S by π/4. That is, theleft eye signal L has amplitude which is √2 times the input signal, andis shifted by ⅛ of the cycle determined by an angular frequency ω in thehorizontal direction. Similarly, the right signal R has amplitude whichis √2 times the input signal, and is shifted by ⅛ of the cycle in thehorizontal direction. In this way, a phase difference of π/2 isgenerated between the left eye signal L and the right eye signal R, thephase difference is perceived as the retinal image difference, therebymaking it possible to sense the depth.

As described above, the retinal image difference is changed depending onthe angular frequency ω. FIG. 14 illustrates a waveform in a case wherethe angular frequency of the input signal is ½, compared with the caseshown in FIG. 13. As understood from the figure, the retinal imagedifference becomes two times larger than that of the case in FIG. 13,and thus, the images are perceived as having a far depth of field in thecase of the binocular stereopsis, compared with the input signal in FIG.13.

Further, FIG. 15 illustrates a waveform in a case where the angularfrequency of the input signal is two times, compared with the case shownin FIG. 13. As understood from the figure, the retinal image differenceis ½ of the case in FIG. 13, and the images are perceived as having anear depth of field in the case of the binocular stereopsis, comparedwith the input signal in FIG. 13.

Further, a waveform in the case of controlling the amplitude of thedifferential signal D is shown in FIG. 16. FIG. 16 illustrates a casethat the differential signal D is amplified by two times. Here, acontrolled differential signal F is expressed as the following formula(formula 6) for generalization.

F=k cos ωx  (formula 6)

Here, k represents a positive real number.

F corresponds to the above-described disparity emphasis signal Egenerated by the conversion process for the differential signal D.

At this time, the left eye signal L and the right eye signal R areexpressed as the following formula 7 and formula 8.

L=S+F=sin ωx+k cos ωx=√{square root over (1+k ²)} sin(ωx+α)  (formula 7)

R=S−F=sin ωx+k cos ωx=√{square root over (1+k ²)} sin(ωx+α)  (formula 8)

Here, α is in the range of 0 to π/2, and is expressed as the followingformula (formula 9).

$\begin{matrix}{\alpha = {{arc}\; \cos \frac{1}{\sqrt{1 + k^{2}}}}} & \left( {{formula}\mspace{14mu} 9} \right)\end{matrix}$

In the formula (formula 9), if the amplified value k of the differentialsignal is increased, a becomes larger, and thus, the phase differencesbetween the input signal S and the left eye signal L and between theinput signal S and the right eye signal R also become larger.Accordingly, the phase difference between the left eye signal L and theright eye signal R becomes larger, and thus, the retinal imagedifference is perceived as large. As a result, the images are perceivedas being further away in the case of the binocular stereopsis.

As described above, in the right eye image and the left eye imagegenerated by the image processing apparatus 100 according to the presentembodiment, the retinal image difference is changed according to thespace frequency of the images. That is, the retinal image differencebecomes small in a region having a high space frequency, and becomeslarge in a region having a low space frequency. In a case where theseimages are separately provided to a person's the right eye and the lefteye, the person perceives the region having a small retinal imagedifference as a near side, and perceives the region having a largeretinal image difference as a far side.

However, the image processing apparatus 100 of the present embodimentsimply performs the process according to a local space frequency asdescribed above, and thus, different retinal image differences are givento an edge section and a texture section for an individual object in theimage. Accordingly, since the observer may be nearly unable to perceivecorrect depth only from the retinal image difference, the overall depthof the image may be analogically perceived by the person from imageinformation about pictorial image characteristics (composition,anteroposterior relation of objects, and space frequency), motionparallax or the like.

Further, as described above, since the retinal image difference ismainly generated for the edge section of the image, the retinal imagedifference is given to minute structures such as the branches of trees,electric cables or hairs. Thus, it is possible to express thestereoscopic effect of minute objects.

The image processing apparatus according to the present embodimentsimply performs a local modulation process for an image using suchcharacteristics, to thereby realize the configuration of generating abinocular disparity image for realizing natural stereopsis.

Further, the image processing apparatus according to the presentembodiment generates the right eye image (Right) and the left eye image(Left) according to the above-described (formula 1). That is, when theluminance level of the video data corresponding to the input signal is(S) and the signal level of the disparity emphasis signal fornon-linearly converting the differential signal (b) shown in FIG. 8 is(E), the right eye image signal (Right) and the left eye image signal(Left) are generated according to the following formula.

Right eye image signal: Right=S−E

Left eye image signal: Left=S+E

As understood from the formula, an addition signal generated by addingthe right eye image signal and the left eye image signal becomes asfollows.

Addition signal=(S+E)+(S−E)=S

As a result, the addition signal becomes equivalent to the input image.

Accordingly, for example, in the case where the display is performed bythe three-dimensional display apparatus of the time division type asdescribed above with reference to FIG. 10 or FIG. 11, if the user who isthe observer observes the image without the liquid crystal shutter-typeglasses, an image in which the left eye image (Left) and the right eyeimage (Right) are integrated by a temporal integration function in avisual system of human is perceived. The image becomes the followingaddition signal [S].

Addition signal=(S+E)+(S−E)=S

That is, an input two-dimensional image can be perceived as it is. Thatis, the image is not observed as unnatural double images, but can beobserved as an image in which no process is performed.

Further, in the case where the display is performed by thethree-dimensional display apparatus of the space division type as shownin FIG. 12, if the observation is performed from such a distance thatone pixel in the vertical direction is not perceived without thepolarization filter, an image to which two pixels in the verticaldirection are added is perceived. The image becomes the followingaddition signal [S].

Addition signal=(S+E)+(S−E)=S

On the other hand, since human eyesight for retinal image differences isten times higher than normal eyesight, the retinal image differencebetween the left eye image and the right eye image can be sufficientlyrecognized even though they are observed from such a distance.Accordingly, the image is not observed as an unnatural double image butcan be observed as an image in which no process is performed, withoutpolarization glasses. With the polarization glasses, thethree-dimensional perception can be achieved.

As described above, the image generated by the image processingapparatus according to the present embodiment is displayed using thethree-dimensional display apparatus, and thus, the three-dimensionalperception can be achieved with the glasses for stereopsis, and theimage can be perceived as an original two-dimensional image in which theconversion is performed without the glasses for stereopsis.

Hereinbefore, in the case where the distance from the photographingposition in the above-described table 1 satisfies “distance>threshold”,the process for the pixel region corresponding to the formula has beendescribed.

Next, in the case of “distance≦threshold”, a process for the pixelregion corresponding to the formula will be described.

As shown in the table 1, in the case of “distance from the photographingposition≦threshold”, the left eye image (L image) is “inputsignal−disparity emphasis signal” (output of subtractor), and the righteye image (R image) is “input signal+disparity emphasis signal” (outputof adder), which are output as the left eye image (L image) and theright eye image (R image).

This process corresponds to a process for switching outputs of the adder134 and the subtractor 135 by the process for switching the left eyeimage (L image) and the right eye image (R image) in the process in thecase where the distance from the photographing position as describedwith reference to FIG. 8 satisfies “distance>threshold”, that is, by theprocess of the selector 137 shown in FIG. 7.

In this case, the right eye signal (c) in FIG. 8 is switched into theleft eye signal, and the left eye signal (d) in FIG. 8 is switched intothe right eye signal. In FIG. 13, the left eye signal and the right eyesignal are switched with each other.

As described above, the phase difference is perceived as the retinalresidual difference, and thus depth is sensed. However, in this case,since the residual difference becomes the exact opposite, compared witha case where switching is not performed, the depth is reversely sensedas near in the case of having been sensed as distant, and the depth issensed as distant in the case of having been sensed as near.

Here, a boundary in which the sensed depth is reversed becomes a pixelin which the distance from the photographing position satisfies“distance=threshold” in which the outputs of the adder 134 and thesubtractor 135 are switched. That is, if the depth is perceived as a farside in the range of “distance>threshold”, the depth is perceived as anear side in the range of “distance<threshold” using the value of“distance=threshold” as the boundary. Thus, it is possible to perceive adepth having a wider range, compared with the process in which theswitching is not performed.

A configuration for such a perception will be described with referenceto FIGS. 17 to 20.

As shown in FIG. 17A, in a case where a non-disparity image in which theleft eye image and the right eye image coincide with each other isdisplayed on the display surface of the display section, the observerperceives the image on the display surface. This is the normaltwo-dimensional image display process.

In a case where the three-dimensional image display process isperformed, the left eye image and the right eye image are shifted anddisplayed. That is, the image with disparity is displayed, and thus, thethree-dimensional perception can be achieved.

For the disparity setting, two different patterns of settings shown inFIGS. 17A, 17B and 18 can be employed. That is, there are two patternsof a disparity setting in FIG. 17B (disparity setting example 1) and adisparity setting in FIG. 18 (disparity setting example 2).

In the disparity setting in FIG. 17B (disparity setting example 1), asshown in the display image in FIG. 17B, the left eye image displayed onthe display surface is shifted to the left side (left side viewed fromthe observer), and the right eye image is shifted to the right side(right side viewed from the observer). In this setting, as shown in theperceived image in FIG. 17B, the observer perceives an objectcorresponding to the image as existing on a rear side (far side) fromthe display surface.

In the disparity setting in FIG. 18 (disparity setting example 2), asshown in the display image in FIG. 18, the left eye image displayed onthe display surface is shifted to the right side (right side viewed fromthe observer), and the right eye image is shifted to the left side (leftside viewed from the observer). In this setting, as shown in theperceived image in FIG. 18, the observer perceives an objectcorresponding to the image as existing on a front side (near side) fromthe display surface.

The image processing apparatus according to the present embodiment has aconfiguration in which both of the disparity setting in FIG. 17B and thedisparity setting in FIG. 18 are selectively applied. That is, the imagesynthesizing section 133 shown in FIGS. 5 and 7 switches the disparitysettings according to the comparison of the distance from thephotographing position and the threshold in the table 1, to therebygenerate the left eye image and the right eye image.

Specifically, the process performed by the image synthesizing section133 is as follows.

With respect to the pixel region in which the determination formula of“distance from the photographing position>threshold” is satisfied, theleft eye image (L image) is “input signal+disparity emphasis signal)(output of the adder”, and the right eye image (R image) is “inputsignal−disparity emphasis signal” (output of the subtractor), which areoutput as the left eye image (L image) and the right eye image (Rimage).

This process corresponds to the process of generating the right eyeimage (c) and the left eye image (d) shown in FIG. 8. This is theprocess corresponding to the disparity setting example 1 in FIG. 17B,and the image is perceived on a rear side (far side) from the displaysurface.

On the other hand, with respect to the pixel region in which thedetermination formula of “distance from the photographingposition≦threshold” is satisfied, the left eye image (L image) is “inputsignal−disparity emphasis signal” (output of the subtractor), and theright eye image (R image) is “input signal+disparity emphasis signal”(output of the adder), which are output as the left eye image (L image)and the right eye image (R image).

This is the process corresponding to the disparity setting example 2 inFIG. 18, and the image is perceived on a front side (near side) of thedisplay surface.

In this process, the right eye image (R image) and the left eye image (Limage) generated by the image synthesis process performed by the imagesynthesizing section 133 are set as shown in FIG. 19.

FIG. 19 illustrates similar signals to FIG. 8, that is, an input signal(a), a differential signal (b), a right eye image signal (c) and a lefteye image signal (d) in the order named from the top.

The input signal (a) and the differential signal (b) are the samesignals as in FIG. 8. The input signal (a) represents change in theluminance of an arbitrary one horizontal line of an arbitrary frame ofvideo data. There is exemplified one line in which a high luminanceregion having a high level of luminance exists in the center areathereof. In a region A ranging from a line position (x1) to a lineposition (x2), the luminance is changed to gradually become high.Between the line position (x2) and a line position (x3), the highluminance region in which the high level of luminance is maintainedexists. Then, in a region B ranging from the line position (x3) to aline position (x4), the luminance is changed to gradually become low.

The differential signal (b) is a result obtained by differentiating theinput signal (a). The differential signal is the signal generated in thedifferentiator 131 of the image converting section 130 in FIG. 5.

As shown in the figure, the differential signal generated by thedifferentiator 131 has a positive value in the region A in which theluminance change of the input signal (a) becomes positive, and has anegative value in the region B in which the luminance change of theinput signal (a) becomes negative.

The right image signal (c) and the left eye image signal (d) becomesignals obtained by switching the right eye image signal (c) and theleft eye image signal (d) in FIG. 8, as described above.

The display process corresponding to the disparity setting example 2 inFIG. 18 is performed by applying these signals, and thus the observerperceives the image on a front side (near side) from the displaysurface.

As described above, the image processing apparatus according to thepresent embodiment selectively applies both the disparity setting inFIG. 17B and the disparity setting in FIG. 18. That is, as shown in thetable 1, the right eye image signal and the left eye image signal inwhich the disparity settings are switched according to the comparison ofthe distance from the photographing position and the threshold, aregenerated.

A process performed by the image processing apparatus according to thepresent embodiment will be described with reference to FIGS. 20A, 20Band 20C as a whole. The image processing apparatus according to thepresent embodiment switches and uses a disparity setting example 2 inwhich an image is perceived on a far side from the display surface asshown in FIG. 20A and a disparity setting example 1 in which an image isperceived on a near side from the display surface as shown in FIG. 20B,according to distance information on pixels for forming the image. As aresult, as shown in FIG. 20C, the observer can perceive the image in awide range from a front side (near side) from the display surface to arear side (far side) from the display surface, to thereby achieve a morethree-dimensional perception.

Next, the sequence of a process performed by the image convertingsection 130 of the image processing apparatus 100 according to anembodiment of the present invention will be described with reference toa flowchart in FIG. 21. The flowchart shown in FIG. 21 illustrates aprocess in a case where the input image is a moving image (video data).

In step S401, the differentiator 131 (see FIG. 5) performs thedifferential process for a luminance signal in video data input to theimage converting section 130. That is, the differential signal (b) inFIG. 8 is generated by the differential process of the input signal (a)in FIG. 8.

In step S402, the non-linear converting section 132 (see FIG. 5)performs the non-linear conversion process for the differential signaloutput from the differentiator 131. This non-linear conversion processis the non-linear conversion process corresponding to the graph as shownin FIG. 6, for example.

The processes in step S403 and below are performed by the imagesynthesizing section 133. In step S403, the controller in the imagesynthesizing section 133 determines whether or not to perform thesynthesis of the left eye image with respect to a current input frame.This determination process is performed according to the display methodof the image display apparatus output by the image processing apparatus100 and a value of a frame counter installed in the image synthesizingsection 133. The frame counter holds a value corresponding to the framenumber of the input image frame.

In a case where the output method of the image display apparatus is thetime division output method shown in FIG. 10, the image synthesizingsection 133 determines whether or not to output the left eye imageaccording to the value of the frame counter. That is, in the case of thetime division output method shown in FIG. 10, the control is performedso that the left eye image is output in either an even frame or an oddframe. If it is determined that the left eye image is to be outputaccording to the value of the frame counter, the procedure goes to stepS404. On the other hand, if it is determined that the right eye image isto be output according to the value of the frame counter, the proceduregoes to step S405.

Further, in other cases which are different from the time divisionoutput method shown in FIG. 10, that is, in the case of the timedivision output method using the two-time frame rate shown in FIG. 11,in the case of the space division output method shown in FIG. 12, or ina case where the left eye image and the right eye image shown in FIG. 9are input to perform display control in the image display apparatus, theimage synthesizing section 133 determines that the left eye images aresynthesized for all the input frames, and then the procedure goes tostep S404.

In step S404, the image synthesizing section 133 generates the left eyeimage (Left). FIG. 22 illustrates a detailed flow of the generationprocess of the left eye image (Left) in step S404. The imagesynthesizing section 133 generates the left eye image (Left) accordingto the table 1 as described above.

That is, the distance from the photographing position is compared withthe preset threshold in the unit of each pixel or in preset regionunits, to thereby generate the left eye image (Left) according to thecomparison result as follows.

In the case of “distance from the photographing position≦threshold”, theleft eye image (L image) is generated as “input signal−disparityemphasis signal”, and in the case of “distance from the photographingposition>threshold”, the left eye image (L image) is generated as “inputsignal+disparity emphasis signal”, through these processes.

Details of the generation process of the left eye image (Left) in stepS404 will be described with reference to a flowchart in FIG. 22.Firstly, in step S501, it is determined whether the generation processof the left eye image (Left) is terminated for each pixel. If it isdetermine that the process is terminated for the all pixels, theprocedure goes to step S510. Then, the step S404 is terminated, and thenthe procedure goes to step S406 in FIG. 21.

In step S501, if it is determined that the generation process of theleft eye image (Left) has an unprocessed pixel, the procedure goes tostep S502.

In step S502, the distance from the photographing position is comparedwith the preset threshold in the pixel unit or the preset region unit.

In step S502, if it is determined that the determination formula of“distance from the photographing positionS_threshold” is satisfied (Yesin step S502), the procedure goes to step S503.

In step S503, the left eye image is generated according to the formula,that is, “left eye image (L image)=input signal (S)−disparity emphasissignal (E)”.

The calculation formula corresponds to a process in which the selector137 of the image synthesizing section 133 shown in FIG. 7 selects theoutput of the subtractor 135 and outputs it as the left eye image (Limage).

On the other hand, in step S502, if it is determined that thedetermination formula of “distance from the photographingposition≦threshold” is not satisfied (No in step S502), the proceduregoes to step S504.

In step S504, the left eye image is generated according to the formula,that is, “left eye image (L image)=input signal (S)+disparity emphasissignal (E)”.

The calculation formula corresponds to a process in which the selector137 of the image synthesizing section 133 shown in FIG. 7 selects theoutput of the adder 134 and outputs it as the left eye image (L image).

On the other hand, in step S403 shown in FIG. 21, if it is determinedthat the left eye image is not synthesized for the current input frame,the procedure goes to step S405, and the right eye image is generatedfor the current input frame.

In step S405, the image synthesizing section 133 generates the right eyeimage (Right). FIG. 23 illustrates a detailed flow of the generationprocess of the right eye image (Right) in step S405. The imagesynthesizing section 133 generates the right eye image (Right) accordingto the table 1 as described above.

That is, the distance from the photographing position is compared withthe preset threshold in the pixel unit or the preset region unit, tothereby generate the right eye image (Right) according to the comparisonresult, as follows.

In the case of “distance from the photographing position≦threshold”, theright eye image (R image) is generated as “input signal+disparityemphasis signal”, and in the case of “distance from the photographingposition>threshold”, the right eye image (R image) is generated as“input signal−disparity emphasis signal”, through these processes.

Details of the generation process of the right eye image (Right) in stepS405 will be described with reference to a flowchart in FIG. 23.Firstly, in step S521, it is determined whether the generation processof the right eye image (Right) is terminated for each pixel. If it isdetermined that the process is terminated for the all pixels, theprocedure goes to step S530. Then, the step S405 is terminated, and thenthe procedure goes to step S407 in FIG. 21.

In step S521, if it is determined that the generation process of theright eye image (Right) has an unprocessed pixel, the procedure goes tostep S522.

In step S522, the distance from the photographing position is comparedwith the preset threshold in the pixel unit or the preset region unit.

In step S522, if it is determined that the determination formula of“distance from the photographing position≦threshold” is satisfied (Yesin step S522), the procedure goes to step S523.

In step S523, the right eye image is generated according to the formula,that is, “right eye image (R image)=input signal (S)+disparity emphasissignal (E)”.

The calculation formula corresponds to a process in which the selector137 of the image synthesizing section 133 shown in FIG. 7 selects theoutput of the adder 134 and outputs it as the right eye image (R image).

On the other hand, in step S522, if it is determined that thedetermination formula of “distance from the photographingposition≦threshold” is not satisfied (No in step S522), the proceduregoes to step S524.

In step S524, the right eye image is generated according to the formula,that is, “right eye image (R image)=input signal (S)−disparity emphasissignal (E)”.

The calculation formula corresponds to a process in which the selector137 of the image synthesizing section 133 shown in FIG. 7 selects theoutput of the subtractor 135 and outputs it as the right eye image (Rimage).

Further, if the generation of the left eye image is terminated in stepS404 in FIG. 21, it is determined whether the right eye image is alsogenerated for the same frame as the generation frame of the left eyeimage in step S406. In a case where the output method of the imageprocessing apparatus is the time division output method shown in FIG.10, since only one of the left eye image and the right eye image issynthesized in each frame, it is determined that the right eye image isnot generated, and the procedure goes to step S407.

Further, in other cases which are different from the time divisionoutput method shown in FIG. 10, that is, in the case of the timedivision output method using the two-time frame rate shown in FIG. 11,in the case of the space division output method shown in FIG. 12, or ina case where the left eye image and the right eye image shown in FIG. 9are input to perform display control in the image display apparatus, theimage synthesizing section 133 determines that the right eye images aresynthesized for all the input frames, and then the procedure goes tostep S405. The process in step S405 is the same as described above,which corresponds to the generation process of the right eye imageaccording to the table 1.

In step S407, the controller of the image synthesizing section 133determines whether or not to perform a reduction process for the image.In a case where the output format of the image processing apparatus isthe space division output method shown in FIG. 12, it is determined toperform the image reduction process, and then the procedure goes to stepS408. In a case where the output format of the image processingapparatus is different from the space division output method shown inFIG. 12, that is, in the case of any one of the methods ofsimultaneously outputting the left eye image and the right eye imageshown in FIG. 9, the time division output method shown in FIG. 10, andthe time division output method using the two-time frame rate shown inFIG. 11, the image reduction process is unnecessary, and thus theprocedure goes to step S410.

In steps S408 and S409, the image synthesizing section 133 generates thebinocular disparity image (e) shown in FIG. 12, as described withreference to FIG. 12, from the right eye image (c) and the left eyeimage (d). That is, the phase of each of the right eye image (c) and theleft eye image (d) is shifted by one line to perform the reductionprocess to ½ in the vertical direction (S408). Further, the imagesynthesizing section 133 alternately synthesizes the left eye image andthe right eye image obtained in this manner in the unit of horizontallines to generate one sheet of binocular disparity image (d) (S409).

In step S410, it is determined whether the image output process in theimage output section 150 is terminated. In a case where the image outputprocess is terminated, the image conversion process is terminated. In acase where the image output process is not terminated, the proceduregoes to step S411.

In step S411, the frame count is incremented, and then the proceduregoes to step S401. Thereafter, the processes of steps S401 to 5410 arerepeated.

Hereinbefore, the example in which the right eye image and the left eyeimage are generated using the input signal (S) and the disparityemphasis signal (E) has been described with reference to FIGS. 21 to 23.However, as described with reference to FIG. 5, the present embodimentmay have a configuration in which the conversion process of thenon-linear converting section 132 is omitted, the differential signalgenerated by the differentiator 131 is directly input to the imagesynthesizing section 133 (dashed line in FIG. 5), and thus, the imagesynthesizing section 133 generates the left eye image and the right eyeimage using the differential signal. In this case, the disparityemphasis signal (E) is replaced by the differential signal.

As described above, according to the image processing apparatusaccording to an embodiment of the invention, the edge section which isthe characteristic amount of the image, that is, the luminance changesection of the image is extracted on the basis of the inputtwo-dimensional image data, and the image pattern of the edge section ischanged, to thereby generate the pseudo right and left eye images. Withsuch a configuration, it is possible to generate a preferable binoculardisparity image in the three-dimensional display apparatus.

Further, in the image processing apparatus according to the presentembodiment, as described above with reference to the table 1, using theinput signal (S) and the disparity emphasis signal (E), the right eyeimage signal and the left eye image signal are generated according tothe following formulas.

In the case of the pixel region in which the determination formula of“distance from the photographing position≦threshold” is satisfied, theformulas are as follows.

Left eye image signal: Left=S−E

Right eye image signal: Right=S+E

In the case of the pixel region in which the determination formula of“distance from the photographing position>threshold” is satisfied, theformulas are as follows.

Left eye image signal: Left=S+E

Right eye image signal: Right=S−E

As understood from the generation formulas of the left eye image signaland the right eye image signal, an addition signal generated by addingthe right eye image and the left eye image signal is as follows.

Addition signal=(S+E)+(S−E)=S

As obvious from the formula, the addition signal is equivalent to theinput image.

In this way, the addition signal is set to be equivalent orapproximately equivalent to the input signal. Accordingly, when the userviews the image displayed on the three-dimensional display apparatus,the user can perceive the three-dimensional expression with the glassesfor stereopsis, and can perceive the image as a normal two-dimensionalimage without the glasses for stereopsis. That is, regardless of thepresence or absence of the glasses, the user can enjoy the image.Further, according to the image converting apparatus according to thepresent embodiment, the disparity of the left eye image and the righteye image is very small, to thereby making it possible to decrease thedegree of fatigue of the user using the glasses for stereopsis.

The image processing apparatus shown in FIG. 1 has been described as animage processing apparatus which does not have the image displaysection. However, the image processing apparatus may be provided as animage processing apparatus having the image display section. FIG. 24 isa diagram illustrating an embodiment of the image processing apparatushaving the image display section.

An image display apparatus 300 receives a still image file output from adigital still camera or the like, or moving image data output from acamcorder or the like, and then converts the received file or data intoan internal data format, in an image input section 310. Here, theinternal data format refers to baseband moving image data, which isvideo data of three primary colors of red (R), green (G) and blue (B) orvideo data of luminance (Y) and color difference (Cb, Cr). The internaldata format may have any color space as long as color space identifyingsignals are overlapped and a color space converting section 32 in asubsequent stage corresponds thereto.

The video data output from the image input section 310 is input to thecolor space converting section 320, and is converted into a luminancesignal and a color difference signal. At this time, in a case where theinput video data is based on Y, Cb and Cr color spaces, the color spaceconverting section 320 outputs the video data without conversion intothe color spaces. In a case where the input video data is based on theR, G and B color spaces, or other color spaces, the color spaceconverting section 320 converts the video data into the luminance (Y)and color difference (Cb, Cr) signals for output.

In this respect, the color spaces of the video data output from thecolor space converting section 320 are not limited to the Y, Cb and Crspaces, but may be any spaces as long as the color spaces allows aluminance component and a color component to be separated from eachother.

The video data output from the color space converting section 320 isinput to an image converting section 330. The image converting section330 generates binocular disparity images for the left eye and the righteye through the process as described above, and synthesizes and outputsthe images according to the format of the image display section 350.

The video data output from the image converting section 330 is input toa color space reverse converting section 340, and is converted into theR, G and B color spaces from the Y, Cb and Cr color spaces.

The video data output from the color space reverse converting section340 is input to an image display section 350. The image display section350 is configured to serve as an image output section and a displaysection, and performs image display according to any three-dimensionaldisplay method (time division method or space division method) asfollows.

(Time Division Type)

In the three-dimensional display method of the space division type, anodd frame and an even frame of input video data are recognized as a lefteye image and a right eye image (or the right eye image and the left eyeimage), respectively, and the images are temporally and alternatelyprovided to the left eye and the right eye by controlling the liquidcrystal shutter-type glasses which a user wears. In this display type,the image display section 350 performs control so that an outputswitching timing of the left eye image and the right eye image issynchronized with shutter switching of left and right glasses sectionsof the glasses used by the observer.

(Space Division Type)

In the three-dimensional display method of the space division type, in acase where a polarization filter which is set so that the polarizingdirection becomes different for every horizontal line is installed on afront display surface and the user uses the glasses of the polarizingfilter type, the images are separated and provided to the left eye andthe right eye for every horizontal line.

As described above, according to the image processing apparatus of thepresent embodiment, the right eye image and the left eye image aregenerated from the characteristic amount of the image in a pseudo manneron the basis of the input two-dimensional image data, and thus, thethree-dimensional display can be performed using the binoculardisparity. Further, according to the image processing apparatus of thepresent embodiment, the images are converted to be equivalent to theinput image when the left eye image and the right eye image are added,and thus, it is possible to perceive the three-dimensional expressionwith the glasses for stereopsis, and to perceive the images as the usualtwo-dimensional image without the glasses for stereopsis. Thus, the usercan enjoy the image regardless of the presence or absence of theglasses. Further, according to the image display apparatus of thepresent embodiment, the disparity of the left eye image and the righteye image is very small, and thus the degree of fatigue when the useruses the glasses for stereopsis can be reduced.

Hereinbefore, the invention has been described with reference to thespecific embodiments. However, it is obvious that those skilled in theart can make modifications or substitutions of the embodiments withoutdeparting from the spirit of the invention. That is, the inventionshould not be interpreted to be limited to the exemplary embodiments.The scope of the invention should be defined with reference to theaccompanying claims.

Further, the series of processes described in the above description maybe performed by hardware, software or a combination thereof. In the casewhere the process is performed by the software, a program in which aprocess sequence is recorded may be installed and executed in a memoryin a computer which is assembled in specially used hardware, or may beinstalled and executed in a general-purpose computer which is capable ofperforming a variety of processes. For example, the program may berecorded in a recording medium in advance. The program may be installedto the computer from a recording medium, may be received through anetwork such as LAN (Local Area Network) or the Internet, or may beinstalled in a recording medium such as a built-in hard disc.

The variety of processes as described above may be performed in thedescribed order in a time series manner, or may be performed in parallelor individually according to a processing ability of a processingapparatus or as necessary. Further, the system in the embodiments has aconfiguration that a plurality of apparatuses is logically combined, andis not limited to a configuration that respective apparatuses areinstalled inside of the same casing.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2009-247931 filedin the Japan Patent Office on Oct. 28, 2009, and Japanese PriorityPatent Application JP 2010-196486 filed in the Japan Patent Office onSep. 2, 2010, the entire contents of which are hereby incorporated byreference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

1. An image processing apparatus comprising: an image input sectionthrough which a two-dimensional image signal is input; an imageconverting section which receives the image signal output from the imageinput section and generates and outputs a left eye image and a right eyeimage for realizing stereopsis; and an image output section whichoutputs the left eye image and the right eye image output from the imageconverting section, wherein the image converting section is configuredto extract a spatial characteristic amount of the input image signal andto generate at least one of the left eye image and the right eye imagethrough an image conversion process in which the characteristic amountis applied, and performs the image conversion process of a differenttype according to a comparison result of distance information of apreset region unit in the input image signal and a predeterminedthreshold.
 2. The apparatus according to claim 1, wherein the imageconverting section is configured to extract a luminance differentialsignal of the input image signal, to set the luminance differentialsignal as the characteristic amount, to generate one of a convertedsignal obtained by adding the characteristic amount to the input imagesignal and a converted signal obtained by subtracting the characteristicamount from the input image signal as one of the left eye image and theright eye image, and to output a non-converted signal from the inputimage signal without being processed as the other one of the left eyeimage and the right eye image.
 3. The apparatus according to claim 1,wherein the image converting section is configured to extract aluminance differential signal of the input image signal, to set theluminance differential signal as the characteristic amount, to generatea signal obtained by adding the characteristic amount to the input imagesignal and a signal obtained by subtracting the characteristic amountfrom the input image signal, and to generate a set of the two signals asa set of the left eye image and the right eye image.
 4. The apparatusaccording to claim 1, wherein the image converting section is configuredto extract a luminance differential signal of the input image signal, toset a signal generated by non-linearly converting the luminancedifferential signal as the characteristic amount, to generate a signalobtained by adding the characteristic amount to the input image signalor a signal obtained by subtracting the characteristic amount from theinput image signal, and to generate one of the generated signals as theleft eye image or the right eye image.
 5. The apparatus according to anyone of claims 1 to 4, wherein the image converting section is configuredto switch the process of generating the left eye image or the right eyeimage by adding the characteristic amount to the input image signal andthe process of generating the left eye image or the right eye image bysubtracting the characteristic amount from the input image signal,according to the comparison result of the distance information of thepreset region unit in the input image signal and the predeterminedthreshold.
 6. The apparatus according to any one of claims 1 to 5,wherein the image converting section is configured to generate the lefteye image as a signal obtained by subtracting the characteristic amountfrom the input image signal and generate the right eye image as a signalobtained by adding the characteristic amount to the input image signalin a case where the relation between the distance of the preset regionunit in the input image signal and the predetermined threshold satisfies“distance≦threshold”, and to generate the left eye image as the signalobtained by adding the characteristic amount to the input image signaland generate the right eye image as the signal obtained by subtractingthe characteristic amount from the input image signal in a case wherethe relation satisfies “distance>threshold”.
 7. The apparatus accordingto any one of claims 1 to 6, wherein the image converting sectionswitches an addition process and a subtraction process between the inputimage signal and the characteristic amount according to the distanceinformation of the preset region unit and generates the left eye imageand the right eye image of which a perception range is enlarged toanteroposterior regions of a display section.
 8. The apparatus accordingto claim 1, wherein the image converting section is configured togenerate the left eye image and the right eye image for each frameforming a moving image.
 9. The apparatus according to claim 8, furthercomprising the image output section which outputs the left eye image andthe right eye image generated by the image converting section, whereinthe image output section is configured to alternately output the lefteye image and the right eye image generated by the image convertingsection at a speed two times faster than an input image frame rate. 10.The apparatus according to claim 1, wherein the image converting sectionis configured to alternately generate only any one of the left eye imageand the right eye image for each frame forming a moving image.
 11. Theapparatus according to claim 1, wherein the image converting section isconfigured to generate the left eye image and the right eye image foreach frame forming a moving image and to generate a binocular disparityimage alternately including line data forming the generated left eyeimage and right eye image.
 12. The apparatus according to claim 1,wherein the image converting section is configured to generate the lefteye image and the right eye image so that an addition signal of thegenerated left eye image and right eye image is set to be equivalent orapproximately equivalent to the input signal.
 13. The apparatusaccording to any one of claims 1 to 12, further comprising an imagedisplay section which displays the images generated by the imageconverting section.
 14. The apparatus according to claim 13, wherein theimage display section is configured to perform a three-dimensionaldisplay process using a time-division type in which the left eye imageand the right eye image are alternately output.
 15. The apparatusaccording to claim 14, wherein the image display section is configuredto perform display switching so that an output switching timing of theleft eye image and the right eye image is synchronized with shutterswitching of left and right glasses sections of glasses which an imageobserver wears, when performing the three-dimensional display processusing the time-division type in which the left eye image and the righteye image are alternately output.
 16. The apparatus according to claim13, wherein the image display section has a configuration in which apolarizing filter which is set so that a polarization direction becomesdifferent for every horizontal line is installed on a front surface ofthe display section, and is configured to display a binocular disparityimage alternately including line data forming the left eye image and theright eye image generated by the image converting section.
 17. An imageprocessing apparatus comprising: an image input section through which atwo-dimensional image signal is input; an image converting section whichreceives the image signal output from the image input section andgenerates and outputs a left eye image or a right eye image forrealizing stereopsis; and an image output section which outputs the lefteye image and the right eye image output from the image convertingsection, wherein the image converting section is configured to generatethe left eye image and the right eye image so that an addition signal ofthe generated left eye image and right eye image is set to be equivalentor approximately equivalent to the input signal, and performs an imageconversion process of a different type according to a comparison resultof distance information of a preset region unit in the input imagesignal and a predetermined threshold.
 18. An image processing method inan image processing apparatus, comprising the steps of: inputting atwo-dimensional image signal, through an image input section; receivingthe image signal output from the image input section and generating andoutputting a left eye image and a right eye image for realizingbinocular stereopsis, by an image converting section; and outputting theleft eye image and the right eye image output from the image convertingsection, through an image output section, wherein the step performed bythe image converting section includes: extracting a spatialcharacteristic amount of the input image signal; generating at least oneof the left eye image and the right eye image through an imageconversion process for performing an emphasis process in which thecharacteristic amount is applied to the input image signal; andperforming the image conversion process of a different type according toa comparison result of distance information of a preset region unit inthe input image signal and a predetermined threshold.
 19. A programwhich allows image processing to be executed in an image processingapparatus, comprising the steps of: enabling an image input section toreceive an input of a two-dimensional image signal; enabling an imageconverting section to receive the image signal output from the imageinput section and generating and outputting a left eye image and a righteye image for realizing binocular stereopsis; and enabling an imageoutput section to output the left eye image and the right eye imageoutput from the image converting section, wherein the step of enablingthe image converting section includes: extracting a spatialcharacteristic amount of the input image signal; generating at least oneof the left eye image and the right eye image through an imageconversion process for performing an emphasis process in which thecharacteristic amount is applied to the input image signal; andperforming the image conversion process of a different type according toa comparison result of distance information of a preset region unit inthe input image signal and a predetermined threshold.